feat(linux): run the firecracker provider without sudo - #11
Merged
Conversation
Booting a sandbox on Linux took privilege twice per boot, and issue #9 was right that both asks were too large. Creating the bridge and TAPs shelled out to `sudo ip`, where a NOPASSWD entry hands over the host's entire network configuration to save one password prompt. Injecting the worktree loop-mounted the rootfs through `sudo clawk __loop-mount`, a root helper that passed its argv straight to syscall.Mount — the widest privilege in the tree, since anyone who could run it could mount any image anywhere. Neither is necessary. Networking: the kernel hands out CAP_NET_ADMIN for free inside a user namespace you own. The daemon forks an anchor into CLONE_NEWUSER|CLONE_NEWNET with a single-uid mapping (no /etc/subuid, no setuid helper — clawk wants capabilities over its own network, never over other users' files), builds br0 + tap0 + gv0 there with netlink, and passes the gvproxy-side TAP fd back over SCM_RIGHTS. gvproxy stays in the host namespace, where its egress sockets belong; only the VM's NIC moves, entered via nsenter because setns into a netns also demands CAP_SYS_ADMIN in the owning userns. Such a boot performs no privileged operation at all and leaves the host's interface list empty, and the VM loses its view of host networking — a real isolation gain. The anchor blocks on a pipe the daemon holds, so the namespace and every device in it dies with the VM: nothing to leak, nothing to clean up after a crash. Worktree: it becomes its own ext4 disk, built in userspace by the same writer that already builds every rootfs, and clawk-init mounts it from /dev/vdc via an additive Block/FSType transport on manifest mounts. The guest still sees /workspace/<name>, so nothing downstream changes, and the loop-mount helper is deleted outright. Hosts that forbid unprivileged user namespaces (Ubuntu 24.04+ via AppArmor, or a zeroed sysctl) fall back to bridge mode. The CLI provisions those devices in the foreground, while it still has a terminal to authenticate on — the daemon never has one — so sudo prompts at most once per sandbox and never per boot. The mode is decided once and pinned for the daemon, with the reason travelling alongside it so the daemon log names the real cause. doctor reports which mode a host will use and the setting to change; CLAWK_NET_MODE pins either. Also here: one L2 segment and guest MAC per sandbox, so concurrent VMs sharing gvproxy's 192.168.127.2 stop colliding; every host device name scoped by uid, so two users' identically-named sandboxes cannot touch each other's devices; the in-guest binaries shipped prebuilt, so an installed clawk boots a sandbox with no Go toolchain present; the `vm ( kernel … )` override honoured instead of silently dropped; a Linux quick start; and CI publishing linux/amd64 and arm64 binaries. Verified on an x86_64 KVM host: rootless and bridge boots, snapshot/resume, worktree ownership, egress filtering, concurrent sandboxes, and a privilege matrix across four users — no sudo, NOPASSWD for ip only, and password sudo. A user with no sudo rights at all boots rootless with no host devices.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Booting a sandbox on Linux took privilege twice per boot, and #9 was right that both asks were too large:
sudo ip, where aNOPASSWDentry hands over the host's entire network configuration to save one password prompt.sudo clawk __loop-mount, a root helper that passed its argv straight tosyscall.Mount— the widest privilege in the tree, since anyone who could run it could mount any image anywhere.Neither is necessary.
Networking
The kernel hands out
CAP_NET_ADMINfor free inside a user namespace you own. The daemon forks an anchor intoCLONE_NEWUSER|CLONE_NEWNETwith a single-uid mapping (no/etc/subuid, no setuid helper — clawk wants capabilities over its own network, never over other users' files), buildsbr0+tap0+gv0there with netlink, and passes the gvproxy-side TAP fd back overSCM_RIGHTS.gvproxy stays in the host namespace, where its egress sockets belong; only the VM's NIC moves, entered via
nsenterbecausesetnsinto a netns also demandsCAP_SYS_ADMINin the owning userns. Such a boot performs no privileged operation at all and leaves the host's interface list empty — and the VM loses its view of host networking, which is a real isolation gain. The anchor blocks on a pipe the daemon holds, so the namespace and every device in it dies with the VM: nothing to leak, nothing to clean up after a crash.Worktree
It becomes its own ext4 disk, built in userspace by the same writer that already builds every rootfs, and
clawk-initmounts it from/dev/vdcvia an additiveBlock/FSTypetransport on manifest mounts. The guest still sees/workspace/<name>, so nothing downstream changes, and the loop-mount helper is deleted outright.Fallback
Hosts that forbid unprivileged user namespaces (Ubuntu 24.04+ via AppArmor, or a zeroed sysctl) fall back to bridge mode. The CLI provisions those devices in the foreground, while it still has a terminal to authenticate on — the daemon never has one — so sudo prompts at most once per sandbox and never per boot. The mode is decided once and pinned for the daemon, with the reason travelling alongside it so the daemon log names the real cause.
clawk doctorreports which mode a host will use and the setting to change;CLAWK_NET_MODEpins either.Also here
192.168.127.2stop colliding.vm ( kernel … )override honoured instead of silently dropped.linux/amd64+arm64binaries.Upgrading
Every host device name now carries the invoking uid, TAPs included. Sandboxes created by an earlier clawk have differently-named devices, so the first
clawk upafter upgrading reports the expected device as missing —clawk down && clawk upre-provisions it. The old devices are no longer named by anything and outliveclawk destroy; remove them withsudo ip link del clawk<hash>(listed byip link | grep clawk).A worktree disk that fails to mount now fails the boot rather than coming up with an empty workspace and reporting success.
Testing
Both modules build, vet and test clean on
linux/arm64andlinux/amd64. Beyond unit tests, this was exercised on a bare-metal KVM host (Ubuntu 24.04, x86_64, firecracker v1.12), each behaviour change A/B'd against a build of the parent commit:NOPASSWDforiponly, and password-sudo — coveringdoctor's OK/WARN/FAIL verdicts and the sudo-classification paths that only misbehave on an ip-only sudoers./dev/net/tunfd in the daemon, working egress from the guest.macOS is validated by this PR's CI run, not locally — the vz provider is cgo + Virtualization.framework and cannot be cross-compiled, so the
macos-14job is the first real compile of this branch on Darwin. Worth a look before merging.Closes #9